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Abstract 



Imputed Social Cost 

'• -'(*■''■ ■ I- 

V7 



If a, loss f^^tion is available; Specifying .the social cost of an error 
of measurement in the score .on. a unidimensional test an asymptotic method, 

based on it ern^ response theory., is developed for optimal test, design, for a 

■ ■ *- .' . ■_ *« 4 • *i .•.•••* •• * ' * * ••. ' " * . v* • • - • • » « ^ ^ .... 

specified target population of examinees. Since in the real world 

such loss functions are not available, it is. more useful to reverse 

this process's - : « thus' a .method is developed for f inding the loss funct 'o. 

for which a given test is an optimally ' designed test for the target 

population . ^ An illustrative application is presented for one* operational 

test - . J ' . ■ " . * ■ " ' 
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Imr^^d So^i^l' Cost 



Estimating the Imputed Social Cost o £ E 



Edr'a unidlmensional test,' the error * f irem^J, . -t 1 dif- \ V 

f erehce tatweeh' the 'examinees t^uBabilitr^ ^ ? h th ; jsti*-- cfi of this 

ability represented by the exsniWefi'a Cist — . discrepancies 

between 0 and 0 may lead to aironeous Li or.: « ' the. oxaml.ncr. , ^ 

(misclassification, erroneous acceptance oi -u on there 1;; an ,. • ■ ; 
expected social cost associated with any; r „<j .'-aa. , (^6) ■ V^fhis y . - . ; 

cost is given: by * some lo^s^uxict^iort L(0s6) ■ " % , \ ■ - . • 

The obvious problem, here* called Problc=m 1 4 IsVf Given the loss func- ' - . • 
tion L(e,e) , how can we build aaioptltnal-L) n -Itam test that will - . . 
'riinltnlze the expected loss over-* specified target population of examinees, 
subject to certain constraints on the statistical characteristics of the ^ 
Items in. the available item pool? iairig i=am 'response theory , [Lord, ■» •/ . 
1980; Hullnv Drasgow^ and Par 30^, 1983],*= solution of this problem , 
will be given' here for a unidimemslonal tesst;.- *, . ' ' ' » 

Unfortunately;, in practice^tt ii unli-^ely that ^< 6 I 6)^wlll be , 
known to the test designer.. •Sptne^ing of -practical value- can still.be .salvaged, 
however, if we can; deal with 'Wt»l« ? i 6 an existing unidimensional ; 

:t est and a specified target popufa^pf examinees, find the* losV^funo- 
tion L(6,0) for .which t his tept i an op timally designed test. If the 



-*The theoretical work iti .Sections -1-4— . was supported b y contract 
N0001 4-80-0=0402, project desigflatlort NR ^ 5 0 =453 between the Office of 
Naval 'Research and Educatipnai Tasting Service, , The empirical, work, 
using ETS data , was 'supported.'** .HS fund« . The wri ter is very much 
indebted to- Martha L . Stockit,g, wko was responsible for obtaining, the 
empirical results reported . in section 5 . . ' • ' , ; 
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lo As function found for . Problem 2 does not agree with our intuitive 
notion^ ag to what is appropriate/ we will" probably redesign future 
test farms to avoid ; this discrepancy- . ' \ . ^ . 

In order to solve Problem 2 s it is necessary first to solve Problem 1; 
this is done in the fi,rst section. The solution to Problem 2 is outlined 
in the second section- Imparlance under trah's format ions of the ability 
scale is discussed in Section 3. In Section 4,. a method for estimating 
the ability distribution of f the target population is discussed - 
An 1 illustrative application to an actual .test is given . in Section 5 _ The 
final section briefly discusses some implications for optimal test 
: design/ ; ' . . ; " .' . V - . , * ■ 

It is assumed here that all item parameters have been determin<=d ^ 

- ••'?**• * , * '• ■ ' . ' ■*■ 1 '• ** -■ • ■ " « * ■» 

by pretesting to sufficient accuracy so that they can be treated as ' ■ ..' 

. - - ' .;■ ; > : ' . ■ . [ ■ ' " * "'. ". : ' , ' ' ■ . 

k\iown. The illustrative example and some d the discussion are bas^d on 

the three-parameter logistic model of., the item response function, (w^-th 

which the reader is assume d^^a be familiar), but. the^proof s of the. main 

results are much 'more general. The examinee's "actual score 6 is 

■ ■ & . . ■ ■ • . . . . 

assumed to be the maximum likelihood estimate of 8 ,= calculated from 

the examinee r s responses to the n test IMS, * . ' 

1. Minimising Expected Loss ^ ; 

For a group of examinees at a^jLven ability level 8 f : the conditional 
expected loss is by definition * »■"'... • •••• :: " '• 



.ImpUted' Social Cost;-. ; 



*'ia|e) = /,:LCe»e),;:4(e.|e) de 



where >(e|6:) .Is the conditional distribution pr;e : ; and J. denotes expecta- 
tion. , If the distabutibn of sbUity 8, in the- target -popuiation is denoted 
by g(e) , then the overall' (unconditional) expected .loss irby def initibn 



This 



■ • g(p / 6Ct fe) i (|) i& .'• t • ; v . 

is the quantity to be .minimized . by optimal test design. 



(2) 



Loss Function* 



Certain reasonably assumptions will be made about the loss function: 
1. • 1.(0,0) - p^^ause *j!!^-e - - 6 , thesis" no irrWofj measure- 
ment and hence no' loss due to- error of rieasurement),, : 



2, When 6 8 , • 1.(0,0) > 0 , 

3, When 5 is near ' 6. , the loss function, and its fWit^two'- 
derivatives with respect to S are continuous, theythird' 

^le^J^s"Sunded. [These conditions will guarantee the . 

convergence of (3) . 1 ' _ 
%. The loss function doss no'r change too sharply with changes in 

' 0 (as will be discussed ia-ter) , ■ , *' • 

For fixed 0 ', expand 1(6,8) in powers'of ■■ 0. - 0 , obtaining 



: i.(<r,o) - i,(o,o) + (o - 



i)l ' (o ,0) -i- | (o - o)V (e,ej + 



8 
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5 



whet C L' (8,6) and T."((),0) denote successive derivatives .of L (-8, 6) 

tfit h raipect L e 8 . evaluated at 6 - (J .* The first term vanishes 

becft -use thete :is no error of measurement when 8=6,, The second . 

tefin^ ^vanishes because for fixed 8 , ^(e;,6) has a; minimum at "8 - 6 . 

: L(-6^8") = i (6 >: 0) 2 L- w (6';.0) plus higher order terms-. (3) 



Higher power-p oE ■ (6 - 0) can be neglected if n is not' too small, 
r.tnr- o. 6 0 in probability as n ► ■ [Lord; • 1980, "p. 59]. 

When £3) is ^substituted into (1) , L"(e ,e1 comes out froin under the 
integration- s^-gn . It, is then apparent that asymptotically (that is, 
for large . u ^3 ■ ■/ 

- . IfV ■- i L n (e*e.) var(eie) ' . 

in rxtem response theory, the asymptotic (conditional) variance of 6 * 
is c^he reciprocal of the test information function 1(9) , [Lord;, ,1980, 
Spcttion 5 •3] * Tfius we shall rewrite the expected loss (2) a£ 

p _ . . : ; . .= "..■ . *' . 

* ' • , % ■ ,*.'•" 

Itilc?_rm a eion F ^unct ion .' " : ■ • • :■ 

■■ ■ j ' " • . ■ - . 

- The iteia. _ response function. P ± ' = P ± (e) is the probability of a 

eor^ect respe* xisfe «to> item i by a randomly chosen examinee at ability 
lev^sl -9 , T lie information function is , 



' 1% 
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Key. -: 



Var(0|0) 



(6) 
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wh< 



ere Q ± ;=- 1 - p ± and P| 5 ~dP ± /d ( 



1 Ordinarily jP. depends on an. item difficulty parameter h ± 



Furthermore, h, is typically simply a translation- parameter! 



it 



b , In this standard 

3. 



:e 6 - b J 



affects only through the difference 

situation, ,b ± also, affects'" P^ only through the difference v - ~ ± 
Thus the area under any function F of P and ' P' pveriha whole 
range of 0 ' ' '* , ■;■ , f 



/' F(&.- b ± ) dO = , ./. F(0) d6 



.is Independent of " b^ . ..' The area under the test information function' 
thus doei not depend 'on b / in these typical models, which will be 

'assumed here. > , ;• y"' >' . V. ; • 

In the special case where P ± (6) ' '±6 the ? three-p^rameter logistic 

"function 7." » • ;»;»,,- . '■ , " 1 ■ 



7 



1 - 'e. 



l\ (0) - e, + 



1 1 + exp[Tj-.7a 1 Ce - b^I ' 



(7) 



we have 



1.7i 



Q.(?/ - c,) 



i l - c. i- i .-. 1 



f 10. 
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and 



^ : '.• . » P [2 ••..pj dP ± ;i. ?! 

1(8) de - / ' >: = 2 '' ^•.• , 07 : -Ie de " ! : / P7Q t i 



n 1.7a, . , . .'" ' ' "'• ' . : 

- Y. ; — J - (1 - c, t- c log c.) • , "W ~ V 

■ jt-r 1 = c i 1 1 " • ' , • • 

This area does not depend oh . . » * , > . >• 

Test Constraints ■ , 

there are always, constraints on a the availability of Items for test 
construction. Item writers can control to a considerable ■ extent the 
difficulty level of the items they write. The discriminating power of 
the available items, however, can. increased only by writing: 

more items and thon discard inf. a larger ^percentage of the items written— 

an expensive procedure. 

it will he assumed here that the test developer has available, an , , 
uri^Ued pool of Items at whatever difficulty levels he or she may specify. ^ 
. The iteins in the pool have already been pretested; faulty; items. 
V especially those with low discriminating power J have already been dis- 
'/carded. The test developer is to build parallel f orms, of a test from 

the item pool; . selecting items Only on the basis of their difficulty b { ,,; 
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* so that each parallel form has the same distribution of h 4 * I turns' 
--cannot be selected on the basis .of their discriminating power , since all fc - 

=' •" • •' . . " • • . •• ■ \ " ; ; " 

items not discarded after pretesting must eventually be used-. In the actual 

* ■'■ . • ■{ : - ' ■ . ' ■ '• "' V V • V'" ' • ' '■■ '"' , V- V- '.. 

test produced, the ^frequency distribution- of other item parameters,- such as 

item, discriminating power, is to be the same- as in the total pool of pre- 
tested items. It will be assumed here - that in the item" pool the distribution, 
of other, item parameters is independent of the 'item difficulty b^ '■■ This ? t ., - 
assumption should be checked empirically for any practical application.. V \ 
This Assumption may fail to hold because of the^ essential* naturev of 
. the test Items;' often it alio fails to hold simpl-y bicause "pretest ..item-. ^ 
test biseriais have been used instead of It he IRT discrimination parameter , 
a s to exclude poorly discriminating items from the. available itemS^pool, 

When item-test biseriais are used in this way for mult ipleN^hoice items v , 

*% ' ' ' .'•'•>' ~- _ * ' ■ ^^^^ 

the harder the item, the higher the a ± parameter must, be for the 'Mem 

p ■ - L 

to escape exclusion from the item pool. This ia true because 'among (items 
with identical a, , the more guessing the lower the item-test biserlal, 
It follows from these, assumptions that the total area under tthe 



test information -function is fixed, .the task of the test developer^is to 

minimize* l(L) by choice, of b. ( i ■'■ 1,2.,'-'- • ?n ) ; no other relevant 

variables are available to the test developer for achieving -this minimization. 

• * 5 ■ ■ 

'.-•".= ' - ' ' . • • ^ 

Minimi zgtt ion . 1 •• '.= .■.. / 

By the Caiichy inequality, - \ "' ' ■ 



■ ' ' T ff o > 2 



12 
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Here^ tbi rirst integrdi twice th<i expected Ipps (5) written tn 
aBbreviated notation*- - Transposing^ we have " .r s *v V\ * ^ 



' . In Frpbleni li L" (0 ,0) •■ and g(0) 4 are/ known j fur ther^e. ^t^- ^^M^ 

theie is an r_(0) such that: equality holds in <9)Y , then thjjr is l:he y .; / . 

^that- minirali^sv the expected loss ( r >) . Equality will hold in '•'•"X 
provided • . '- *'..'., .' .'. '-.v . i"'; 

• ".\^S^f ■ 1Je -:v ■> v - . ' ' ..• ' .- v. :.; [ :■.:;' . ; -, •• : •- 

- -1(8) is proportional to /L" (6^9 ) g(9) - " . T 

' Monetary Units . ,' - . . ' . j •• ,-, -vc 

The loss function L((),0) is necessarily expressed in terms of some 

-arbitrary^ 

«#^mit so that the area under" : /[X/ 1 (6 ,0) g (6) ] Is; equal-to: ' C.^ 1 *!*^^ 
this last being a known and fixed quantity determined by * n and by the .. •. 
item parameters,- excluding the # b , of the item pool . Once thi s choice 
of unit has been made, i$e expected 'loss will be minimigai if the ^ ; . i 

•'•Is • -developer can build a test with . " -* " • ; 

% - : : lip). "•/i.'. , .(o,o) g(o) ' ^ . * • (107 : 



S^n.^;- ^;-; r ^.\. -V-:'; A- v..'-""*.- J , Imputed Social Cost , 

\ ' ; r" * ' ' ' ' * 10 . 

A - Building the Test \ ~ s v **r^". _ >,< *\y - z . " 5 ?V 

v.'-vv-;-:'.' ' .-. * ■:' j ,^v ' r - • ' " ■" , . , . \*. ; - J --' .'- .V -...v VV;-"...'-.".' •: [i""'*":..; /'..'v. i 

; ^ M -Birnbaum [1968, Section 20.6] s.uggested an effective cut-and~Cry -^V 

^ T target infbrmatjonj f urict'^h*^ ia; outlined .in f LoM [1980,-. * ; V 

.Section 1 '^.4] = The, method the test •/;* y/ 

# 4 inf o rma t ion function is simply a sum of : the -^inf brraa tion functions > v ^ ! : 'V , a f 
(PV^P Qi) of the 'items includeS in the test , "V . ; v ' ; ' v j ; vU-"'"';- '-r • 



^ The 'method." is effective ; provided the. targe t information curve^ is 
no t top irregular and does not vary too rapidly as a if unction of 
The results obtained Jiere hold under thisr condition. If the; target curve-* 

'. _v .' - r '■ a • - : . ; >' : ■'>* ^ 4 : : ._ ■ ' "•- 

is >tipp itrr it will no t. be, possible, to build a test having the desired 

information function by selecting items on b^ from the available item pool: 

practical Procedure (Summary) v - . \ " J " • ' 

Given L (6 ,8 ) and g.(8) , to build k parallel test forms of* length : 
n that approximately minimize the expected loss : 



■t : 



1: Plot a , and e , - against b f to verify that the distribution 

..• .:, .. i ■ i - i • . " .; : . . - . - _ • 

of a ^ c £ i n tne item pool is approximate^ the same 

/ ; at- all levels of b. , as assumed in the subsection titled 

'.• .* "* \ " ■ 1 » ■• = "• * * = ' ■ » ■ ■ ■ ••■ ■ ^ ■ ■•*•" 

Test Constraints . / 

2. Compute , ! ; , s !\ 



s . ' ■ M ^ P' 2 

»~ i=l ^ i^i 



where = M is the numb et- • of.it ems in a large item pool ; Note, that 
this inteigral ,4oes; no t depend on the dis t b^* in the poo . 



erJc 



w\ ; >' "V 3. ..Choose monetary units so ; that 
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/ /ii."(o,o) }-,(o)i an . 




is #qual.:to 



k / ; i<e). ae 



for Some iriLeger .. k . '. ,-*v _ , ' ' ; ' ;• ... f. 

' ^ Selecting items 'only on their; , .'.use Birnbaum' s method ^ ^ 
^1 &^to/ select Ja.pi^ 

item information functions is aBPSOximately equal to. . 

' /[l"(0,o) b(q)J . '" • ' ]J l 

• ,"f I . ^piA?ldey- the nk selected items into k vtes^orms of n ■ items 
r each, all approximately parallel to each other * . . • ■_ 



2 , ■ The Loss Function for Which a Given Test Is an Optimal Test 



If a given test is an optimal test, then^C 10) holds and 



l Ce,e) g(0 ) 




(ID 



Consequently , the loss function is given approximately *by (3) and (11) 



(12) 



15 
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Y. ... ;.. •/;'.?" C^£~j -v^v : . . • „• -->':'.■; . ... .• c ' * • 

For fixed * 6 , this Is * the equation of a parabola.' VThen; n* .is hot tod 

smalls i will be close to 8 §nd (12) ^11 provide an *adeqtfate ' 

approximation to the los^ function for those values of 6 ; that are likely 

to Ve observed. • -For= • By close to Q , the desired' loss f unction carf- be : t : 

computed from (12) . for any' given test J; provided g (6) ^it„snecif iady^yyy \' ' 

. ♦ ' •_• . 5 ■ - 3. Transformation of the. Scoria Scale , : ,= / 

;%eVling with problems of test design * Consider the social : 60s t iri yrY./ _ v 
dollars of an error of measurement at a" given ability level y If the y, 
error of measurement (the discrepancy between the actual test score and ; ■•/; 
the true ahility of which ^ is 1 an estimate) is- specif ied as a- multiple ; y 
of its standard error, as^ptotically. (for large - n Y the loss in , ' \ 
dollars will be the same, no matter what scale is used for measuring 

ability. . • • ; /. : ■ '■■ / • x : ; : ^ ; • ~ ... • • • • ' .; 

Instead of using the . 6 " scale of abii^€yT^uppose we use the ; v u 
number-right true-score scale, given by the mono tonic continuous . ^ 
transformation . ^ •,- 



C » ' E P ± (e) 



(13) 



The examinee' s obtained score should now be taken to be 



T.-^V'iA-^' T : : ':! : - ; 4"- ; >^ ; :.- ^'^f ^'-^v. " ; ; : : : ,;av:-: ; ^ : '13;v : -?' '-: 7:77- 

(Note that we need to use here . the. maximum likelihood estimator of £ 
defined by- (14), not' the ^examinee's tyuottber of right answers.) If ' , \ r 
9 " differs from 6 by 7,K - times .S.E. (ele) , than, asymptotically,. 

^^^^iV'^^'^-^y K ' tlmco S * E * t5 l * 7 Asymptotically, ,.^ ^ 
K[S.E. (e | Q) J is actually t he same error of measurement <m the. scale 
as K'[S.E. d \ 6) is on the F. scale;; thus ' the social consequences of 

7 this error $111 be the same regardless oivthe scule used. ' ^ jC : -• y'*r ; ; . 

the C scale, the loss function (12) becomes ' • 7 •? • >;"' '■ ' 7;._7 *' ? "■ '■' ' w 

where g( ) (and J( ) denote the. same functions as previously. This 
equation could be used as 1^ stands, but for reasons of symmetry, it. may 
,be preferable ^'expand it jfor f ixed r in pqwers of ; £ - £ - The result 
_ 1b found to be 7 • . • 77 



.•„.;••• -.n^(C£) -2 ,,[0(01 1 df, 1 (c . FJ ;•; \ '"7,.c 

',•'.'* • , {<" ^ -.7 -.*;,. -7' ■" 7 7*>- • •*_ : yk---"- • •". :•" 

Equation (15) Is used 'here to represent tlie loss function when the 
obtained score is " ? rather than ;0 ... This transformat ion has an 7^ 77 
advantage for presenting experimental results, since the number-right ' 
score scale is mbre familiar to us thsn.the 6 scale; •/.; . . 
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' Note apiin that - the actual monetary loss is the same regardless of * 
the scale agalhsV which it is plotted, ; r Th±s irivarisnee makes the* loss 
Jiir^ti^ mu6^ thatt the inf ©rma t£op, 

^n^tiorii ; Expressed on ther; v6 " scale, tSe te%t inf dtmatio^ £ unc^pii ;foi 
§-r*^is- typically a bell-shaded curve; expressed on the . C y sealfe , ; ; 

test&hf prmatlon iunctio 

[Lord v 1980, Chapter : 6] . This lack of invariance makes ifc dif ficult^p 
use the test inforinatl^h function as a convincing basis for test design 



=4: ~ '4,* Est imating the True g (6) 

' 1 . • t ■ '.' r . .. ■. • ■ •• • ••*. • ■ - "■ * • • ,.".*•*•-- v, 

By : Its definition, expected loss (?) requires, specification of : 
the distribution 6 in the? target population. It is important to note 
that the distribution of 0 in the; target population is. not an : adeguare " 
estimate of g (e) ^ the ? true "distribution 

e- contains errors of measurement and thus has a larger variance, than 
e Since g(0> app<*lira in the. denominator of (12), it particularly 

important to estimate ;g(0) as ' accurately r as possible. V ^ " 



V ■ . To obtain the j^eriGal reaults. o^ the true distribution 

of , number-risht true score tl3) ^ ;h^e denoted byt HO , was osf.imat.ud by 
Method 20 [Lord,' 198^^hapter 161, Sirica ' M'M neceasarily , ^ Z^C--) * 
ah estimated lo^er X$aM' for ; ! ■ § waa set ' atf 5,1^ . where "c ± represents 
the estimated c. parameter oi item ' i . for purposes of Sect ion 5 all 



The require* estimate of g(o) for the targ% population was 
obtained from the Method' 20 estimate of f(f,) by. the relation * ^ 

g(eU- f<€) de . • ^ c t ^ , 

The derlv^ive ±ti (16) is the Marivtftivi of (13) , e>timatei^ii practice 
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5. illustrative Example S^^fc 

, . - .; - " v- • • - ' ' • ■■■■ ••. -v\ - ■ • .. /.v. v ; •• ,. • • -V-v', ..^•v.r,.--' 5 - ->-... • . • ;• , >- 

A represent^Lt ^ve ; sample of 19 » 949 examinees tested*^ 3^^-82 was - 
obtained for the SenVa& iense test in Foi^^ 

Basic SlMls vPlaeem^ cbmpeltency-ite^veon 5 *^ of 35 four- 

choicer items requiring the. examinee to distinguish correct from incorrect 
English i^J&tt^ primarily; to assign certain .entering ; 

College student s to remedial English clas ses . ' : .' . ... _ 

• J : • The item responses of al l 19, 949 examinees were analyzed, using ^LpGIST 
to estimate; the Item parameterf of all ;items in the test. The true distri- 
button of e" for the target population was estimated 4 as described in 
Section 4 (for this i .purpose, 1 a response chosen at random f rom^ the f out 
choicii was supplied 'wherever to respond to an , ipem) . 

The test information function (6)4was calculated from the estimated item . : , 
parameters. .Finally ,-the loss function for which the test is an optimally 
designed' : ;$^s^H was estimated by ^12)., ■ k - ,. .' . ; ;. ■•:.;*■ 



" ; , • Imputed Social Cost 

Fitture 1 shows the actual distribution -of- number-ritht scores (fro- 
^uencyi polygon)^ the ^ numher^ight l^ue^scere distribution estimated by 
Method 20 (solid curve) i and the' corresponding fitted distribution of 

right answers out of a possible 35 * The chi square ^etweeti observedVand v 
fitted number-right score distributions is at the 86th percentile of the 
chi square distribution with 18 degrees of freedom. In view of the large 
sau^le size ( N. = 19,949 ), ihis seems an ^equat^ : f it r as in indeed sug- 
gested visually by, the' agreement shown, in Figure *1- ' - : : 



The estimated loss function (12) fir wW-ch Vf the test is an optimally de= 

: • . r ( > / '<■-,; r : . ' . , /\ ; . ■■■/■.- _■: \. • v. : - ; ...... „•-. .... 

signed , test is plotted, in Figure 2 against the 8 and 8 Scales* The diree 
t ion of the 6 scale is reversed from the cotivfentional direc tibn in order to 
improve visibility , " Loss is shown- on the vertical scale r In - this and 
the next figure, the parabola for any given 6 is drawn only for 6 values 

within, two standard errors of the true 0 , 

• : • ••.* - r • | . " . ' \ . * *' ; _ .,• _ ■• « -" i ; '. . • . _'. • .•• '. 

/' The figure shows that the Sentence Sense test is built as if it were 
important to, measure accurately at high ability levels as well as at low 
^ability "ievjels: Clearly, this is not-appropriate for a competency t,est-- 
the test should assign high losses to errors of measurement at low ability v 
levels but not at high ability levels* . The more difficult iteimV in the 
test should be replaced by easier items. - 1 ' 

' ' T Tha estimated -loss function (15) \ for which the test is an optimally 
designed test is plotted in Figure 3 against number-right true score and , 



the bottom of the figure tun: in the opposite direction from the scales 
at th^-bottom oJT Figure 2* This plot Is easier to interpret than : Figure^ I 
since we are more accustomed 'to the number-right score scale - than to the, • 
8 stale* A The plot looks' very different f fom Figure 2 because: ; : , 
- ^ a nujber-right score of -34 ;is not :$howri> 

The loss function for this score is rather high and would 
^ obscure top' much of the rest of the figure. r : , . : 

2". A 1 wide range, at the high end of the 8 scale is compressed 
' ". / - into a small range of ^number-right scores, as shown diiAthe : 

following table: . - . ¥ ** , : J 

i V -8 10 : 12 14: : 16 ; IB ; 20 22 r ;^24 ^/^V ^ 28 ; 30 
6 * -iv2 -3.1 -2.4 -1,9 -1.6 -1.3 =1,0 -.7 -,4 -v?-: .1 " ,5^ 

"V ' - ' - ' . ; • \: • - ; ' r ' - * • '\r ' - - . .. - \ * - 

. Again, it appears that the test discriminates at ^Lgtr true score leve. 

where discrimination is not really desired. The loss funq^ion at C ^34 

(not plotted) shows a loss of approximately 100 when £ is two standard • 

errors from t For number-right scores of 30 and below, the shape of 

the loss -function seems very appropriate for a competency test, with very . 

high losses attributed to errors of measurement at low score levels, : - . 
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^ , . - r8 . D is cussion - .- " .■ -:. 

In the case ' of' a minimum competency test,,, the; social losses ari«l n« 
frota errors of measurement will be high for. examinees near the cutting 
sco t e, " which is always near the low end of- the score scale. Social losses 
Will be near zero^ 
about ;iheae f examinees 

' -For a 'college admissions /teat, it would seem reasonable to expect. - • :,- 
that errors, of ^asarCTrt^t^ ii.; the scores Vjib^^g^^bii^l:^ r fut.^^tB^ JAl.it.:-_-;:; ; "_;^^ 
result to^relatively higti social losses, ^Somewhat lower social losses \, 
shouid^be-expected.;t 

students. . "/ " ; : '.r:.;V.':.''.A 

" In thu case of grade, school: tests of ' ability ' or of yb^abulary, it 
'has sometimes; been; argued^that , to be fair,: the standard error of k-,,.:; 
mea.suremo.nt of the tent scoijc should be roughly the same for each ? ^ : 
'individual (see, for example, fluliri at al. , 1983, p. 90); The first 
aif f iculty with this Approach is i' that its impltcatiohs for test design^ 
when thu 1 est -score is ;j are completely different than who n the test .. 
score is 1 /' or simply tho number of right -answers'. . Although equality of ; { .. 
Tstandard errors or roeas»iromont at all. ability levels has" strong intuitive - 
appeal, there is no clear' way to iecide Whether, this equality should. hold ^ 
oh ttie 6 scale , oV on the number-right score scale, or on some other v 
scale. It cannot hold Simultaneously on two dif f erent scales unless 
one" scale is 1 linear transformation of tl\e other. . : ; ^ ; . 
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• In any casa, any goal of equal" standard errors of measurement, at 
different ability levels is completely incompatible with the goal of • 

; minimize loss due to -error s\o£ measurement. If we r 1 

wish^ to minimize social IbsS.*" we must , other things! being equal, _-.'.> , : - 
xaqh ilize pur! it es t development resources so as to measure most accurately* • . 
at those ability levels where the most people are found . We c^^tv 
waste items in order to secure accurate measurement at ability levels ^ 
where only a few people will be affected, unless, of course , there ±s * • '" 

^^very— high-^bss Junction at :these ability levels 
of measurement in sparsely populated stretches of the ability range must- . 
b*» sa^rifired, other things being equal % in order to obtain more ^^^ ^ -. . 
accurate measurement in heavily populated stretches* : _ 1 ^ 

> As a' concrete example , consider a vocabulary test for grade 5 and 
suppose our test is built to minimize over al 1- exp ec ted loss * Suppose also , 
as might be reasonable for such a test, that the expected loss at a fixed- 
ability level is constant across ability levels, so that, by (12) , 



where K is some constant ^ It follows that 
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-\ - 23 . , - " - 

Since . g (6 ) is small for extreme' 9 , the standard error of measurement , 
;-7Var:(e| e); s will :in^ftis i case; te- vsfy - wi^h ;larger, j f or. ax^rilnees with 
extreme G than for examinees with moderate 6 / - Thus in this case ; the I 
goal of equal, standard errors of measurement at all ability levels is - . 
utter^ overall expected loss i - ihXs ±s " 

simply an illustration of the fact; that if we wish to minimize overall . : , 
expected loss, our measurement ef fort must he concentrated on the : -Bub-v-:v::^;^;,-- =;.^ 
ranges- qf ability that aremost highly populated in the target population . * 
J.;'..' To* summarize, in respect^to a unidimensional test : - (^^"^^^^ 

1, * Given the loss function, the distribution of ability in V .-/"' 
target population, f arid certain constraints on the available item pool, 4 

-method has been described ^ 

loss. ; • . V \ j\ _ . ■■- . ■ ■■ ■ ^ . • • -. '■ ■■' x ■ ." % -.' '•" '." j ."; : v.: . 

2, Given a test and also the' distribution of ability in the ^ ; * v/v- 
target population, a method has been described for finding the loss 

- •■ ■ ■ ■ • ' ." • ' ■■ ■'■ -1 • •••••• • •••'•••••«•• • .;.>■=-!.-.•.-..' •:•',-*;*■ . -•■ : . :••»-. 

function for which this test is an optimally designed test given certain v' a 

- constraints on the, ^available Jt em pqoI. ..... ■ ~. : : - % . 

3, r Minimizing social loss is in general incompatible with ; ■• = ••' 
equal measurement accuracy . across examinees., To minimize social ';" ■ . 
loss, measurement accuracy must be High (other things being equal)' , 1 V* - : 
over ability, ranges that are heavily populated, and relatively low - :f ; 
over ranges that are sparsely populated f i ; . = . . . ,v . ; / ,'- 



27 



j fltnpu ted Social Gos t '■ 

:.; --/••/•/.- - ; / v '-- : •"• • ' ; ■ "• ■■ . - 

Reference Note \. .. ,= . . 



: V Princeton, N. J.. l Educational Testing Service, February 1982. 




ERIC 



ERIC 




mputed Social Costr 

Refer 



- , 



Birribaum^A. - Some latent trait models and their uses in inferring 
^ in .examinee 1 s ability. Part 5 of M- Lord and M. R. Noviek, 
> - : Statistical theories of mental test scores . Reading > Masa •-'. •„ 

■V- ---- ,\ : ;> • 

'> : ^ ■ ^Afld t son--Wesltty, 1968. : r "; : \., ' 

Htilin, C. t., Drasp.ow, F . , & Fat.ioils , C , K. item' resp onse -theory. 

IV ; iii. r>' Dow T Jpnes-Irwiii 1983. ;•• / \ ; ■ . — "V;' '. : :\ 



>.T.ot^ practical testing 

— ^ — : — : — — „ — — ; — L>i ^-£.^ -,7 . ^ v . : . % .« • , , :-. _,-.. -„ .. ^- . r ..i.. 

problems , Hiiisaaley NtJ. : Lawrence Lrlbaum Associates, 1980. 



29 



Navy ^ ; 



Dr,, Ed Aiken ■ - " 

Navy Personnel R&D Center: 

San I)iego , CA \ 92152 . : 

Pr * Ar thur^BacHrach-: — ' ; 
Enyirohmentar Stress Program Center 
Navai^Medicai Research Institute 



^^^Navy^Personn«l/^&D ? Center 
San Diego, GA 92152 

1 Liaison Scientist 

Of f ice % of Naval Research 
^ ^-^ Branch Office London 

. 3^ Box 139 : . . ■ ,-. . 

^^Ftb m^YoTkt NY^09510 t 



DISTRIBUTION LIST 



CDR Mike Cur ran ^ 
Office of Naval Research 
800 North Quindy >S t reel- 
Code 270 

Ariingtbn;; VA^v22il7r 




Dr* Tom; Duffy 
Navy Personnel Ri 
- San D iego ~ ; ; ■ CA ^9 2152 

Mr * Mike Durmeyer ; ^ ' 
Ins true t ional Program D ey elopment 
Building 90 ' 

NET-PDCD • -'• -; : : >f ' ■; ■ ;• : V : :\;^;/^- 

Great Lakes. NIC, IL 60088 



Lt. Alexander Bory 
Applied Psychology 
Measurement Division Y 
NAtiRL. , v .v VV ; /;. - 
NAS Fensacola, FL 32508 



Dr. Richard Elster " 
-Departme nt 1 of -Adminis t rat-Lve S cience^ 
Naval Postgraduate School ^ : *" VX::'' : \ 
Monterey, CA:. 93940: ; 



Dr. Pat Federico" - - " 
Code P13 ; • , ... 

Navy Personnel R&_D Center 
"Sa^Dieg©^ 



Dr .' Robert Breaux 
NAVTRAEQUIPCEN 

Code N-095R : ; 
Orlando, FL 32813 

Dr. Robert Carroll 
NAVOP 115 

Washington, DC 20370 

Chi ef of Naval E ducat ion and -- — 

Training Lias on Office 
Air Force Human Resource Laboratory 
Flying Training Division 
Williams Air Force Base, AZ 85224 

Dr , Stanley Collyer V 
Of f ice of Naval Technology . 
800N* Quiricy Street 
Arlington* VA 22217 ^ 



Dr. Cathy Fernandes 

Navy Personnel R&D: Center 

San Diego, CA 92152 

Dr* John Ford - 

Navy Personnel R & D Center 

San Diego, CA 92152 V 



Dr . Jim Hollan 
Code 14 v 



Navy Personnel R & D" Center 
San piego, CA 92152 [ \:> 
X; " ■■ ' \ . ? 

Dr* Ed Hutch ins 
Navy Personnel R & D Center 
San Diego, CA '92152 ■ 



30 



ERLC 



1 : Dr V Norman J * Kerr; ".' ; • : r 

Chief of Naval Technical Training 
Naval Air Station Memphis (75) \ 
Miilitigton s TN , 38054 v . ^ v 

1 Dr . Peter Kincaid , : 

'X? Training Aiia^s ti^ & Evaluation Group 

= Department of the Navy . , 

" Orlando , FL 32813 ; 

1 Dr^ R. W. King 



Director, NavaT E ^ u ^ tion 
: and Training Program ^ \ 

"' v ^Niwi^Trainijife Center Bldg* 90 
i Great Lakes, IL 60088 

1 pr> Leonard Kroeker 

Navy Personnel R & D Center 
-^Ban~ Diegot ~4*H* 2 



T Dr* William L* Maloy (02)- 

Chief of Naval Education and Training 
Naval Air Station 
Pensacola, FL 32508 ." 



% : 1 WSichctiealf Director f , 7^: 

Navy Personnel R&D CMtctrr*. 
, / San ;Diego s CA 9 215 i - . -*\; ' " V 

6 Personnel ^.Training; Research Group 

_ . '. 'V; Code 4#2PT ; /% ' -" -4 . ." • ■ ; ■ - • '■" - " 
Office ofJ Naval Research^ _ ^ 
Arlington, VA^ f 2221«7- ; .- . jf "t^f 

• £ 1 Special Asst . 'ftor- Idueatipiv and* ; 
Training (OF-GIE) 
— -Roon^^^ 



~1^ Dry Kneale Marshall- ~: - ^ :; 

Chairman, Operations Research Dept. 
Naval Post Graduate School 
: Monte^y , CA 93940 : '. - . : 

i Dr - James . McB.ride ^ 
• Navy Personnel R&D Center 
San Diegby CA 92152 

^fl - Dr. William Montague 
^ -^NPRDC Go de 1 3 - ^ . ^ , ^ ^ — :\ _ ^ 
; ^ San Diego,; CA 92152 y. 

1 Mr- William Nordbrock ! 
1032 Fairlawn Avenue 
: - Liberty villa ,/XL 60048, 

"1 Library , Code P201L 7 ; 

Navy Personnel R, & D Center , 
; V , San Diegoy : CA V 92152 V; .,\ 



V Washington, DC r 203 70 

1 LT Frank C v Petho, MSG* USN 

CNET-(N-432) -W : . = ^ yj^l 

NAS " . - T ■• V - ■■*:"-:■ 
Pensacola v FL 32508 • > 



Dr * Bernard Rimlatid (01C) - 
Navy Personnel B. & B Center 
San Diego \ CA v92152 1 ; 



Dr. Carl Ross * \ 
CNET-PDCD. ^ • 

Building 90 t;-' ; 
Great Lakes NTC/IL 



60088 



Dr, Worth Scanland, -Director ■ / 

CNET (N-5) ." "; ' 

NAS ■ - 

Pensacola, FL 32508 ■/ ; 

Dr* Robert G- Smith m 
Office of Chief of Naval Operations 

0P-987H ■ ; " ..'" /: •. . • . j 

Washington, DC 20350 v . ; 

Dr . Alf red F * Smode , Director 
Training Analysis and Evaluation Group 
^Department" of the Navy/ 
Orlando, FL 32813 f =y ;.. ; , . 

Dr. Richard S or ens en - 

Navy Pejrsonnel R & D Center 

San Diego, CA 92152. . [ ; ] / 1 



3.1 



ERIC 



Dr. Frederick Steinheiser; - 

CNO OP 115 , ; . 

Navy Annex " • - 

Arlington, VA 20370 ; ' ; 

Mr- Brad Sympson \ : : ' : ; 

NaVal Personnel} % /&:\b;yCe^to.:^/:; ; .^^- ; -;,5' 
San DiegoV CA 92152 . v", . ; ; — | 

Br* Frank VicinoV / ' - V = t ; ' "- • v 
Navy Personnel R Sr D Center ;: . . -.' . .-' 
•San Di^egoV CA 92 152 ,- ^ • ; : '■^■£,1 ". ' ■ ■ . 

2j)£. Edward Wegmari - "' '.A •- 

Office of Naval- Research (Code -4 11S&P ) 
800 North Quiney Street 

Arlington, Vf 22217 . v >-/ - 

^r^Rona^d^ We itgi^^^-^^-^-^^-'r^ 
Code" 54 ; " ^ - % ^ \ . 

Department of Adtoinistratiye Services 
U-S. Naval Postgraduate School ; 
Monterey, CA : 93940. C • ; 



Dr . Douglas Wetzel 

Code 1 £~ A ^ r ^ 

Navy Personnel R & : D •Center 

San Diego, CA 92152 

Dr . Martin F • V/iskoff l - 
Navy Personnel R & D Center 
San Diego, CA 92152 . 

Mr. John H. Wolfe 
.Navy Personnel R &D Center 
San Diego,: CA 92152/ — . 

Dr * Wallace Wulf eck , III : 
Navy Personnel R & D Center 
San Diego ,/cA 92 ISjZ ;j 



Marine, Corps 



Dr *;\ H V Wi 1 liam G r eenup 
Education Advisor (£031)! 
Education Center , v MCDEC 
Quanticb,:/VAi 22134 1 • 



Director; Off ice of Manpower 
^Utilization • - ; u : - • ' 
HQ 4 Marine Corps (MPU) 
BCB j Buiicfing 2009 

IQjianticov; VA\'" ; 2213^ ; - 

Headquarters y U> S . Marine :CorpSv 
Code MPI-20 < \ 

Washington; ife 

Special Assistant for Marine: f J 
[: Corps Matters; i x[\- / • / 

Code 100M ••: 4^;^. r-;^ -^.::\;:.;;f ";=4 : 
-'■ Of fice of Naval Research " - ;-' 

800 N. Quincy Street . 
_ Ar lingt on, VA- 22217 . v 7 

Dr. A . L> Slaf kosky ' - - / 
^S ci entific^ AdyjL 
Code RD-1 * ' "\ 
HQ i U>S * Marine Corps 
Washington, DC .20380 



Major F r ank Y ohannan , S USMC 
Headquarters , Marine. Corps 
;XCo<ie^MPi--2 0^ ^ _ ^ ... .. _ 
Washington, DC 20380 



: .Army- '• / ■ ; •■'< 

Technical Director 

U.S. Army Research Institute for th 

• Behavioral and Social Sciences 
5 00 1 Eisenhower Avenue -, 
Alexandria, VA 22333 ' 

Mr- James Baker 
U.S. Army Research Institute 
5001 Eisenhower Avenue 
Alexandria, VA 22333 \ ; i, 

Dr Kent Eaton- ^ = • , / ; ; >. 

U,S . Army Research Institute . 
5001 Eisenhower Avenue ; • ^ . p.; 
AleKandria , VA 22333 ; • . ,- : 



y*S* Array Research Institute : t - 

5001 EisenKower' Avenue 

jaeKandtia- VA: 22333 V ; ; . 

1 . Dr / Myron Fiscal^ ; ^ ^ \~. ^ 

U.S . Army;:Researeh Institute for the 
= J Social khd Behavioral Sciences 

5001 Eisenhower Avenue * 
- -Alexandria; VA 22333 -v : ,; .. ; ■ , 

1 Dir. Hilton S* Katz ■ ' :■■ \v : 
:: ; training Technical Area / ^ 

U*5* Army Research Institute ;-' 4 
5001 Eisenhower Avenue 
: Ale^naria, VA 22333 v 

1 Dr . Harold F - 0 f Neil Jr- 
/ Director j Training Research Lab 

^.jj.^L U* _S . Army Research Institute • 

.5001 Eisenhower Avenue , 
Alexandria, VA 22333 V 

1. Commander, U.S. Army Research 

•' -'•■■-.= ■• Institute ..• _ : - ; /• • --• ;. • • • 
" ATTN: 7 PE RI— BR (D r *" J lid i th Orasanu) 
„ 500 1 E i s enhowe r Aven ue . . . 

• Alexandria, VA 22333: ^ "~ -[ 

1 Dr. Joseph Psotka > 

ATTN : PERI-IC . _ 

U ,S * Army Research Institute 

; 5.001 Eisenhower Avenue ; . " ;." .. 

^Alexandria, VA 22333- 

1 Mr. Robert Ross 
^ U.S.. _ A rmjrjRe e ; ear eh insti tut e for ^ 
the Soeiai and Behavioral Sciences 
' 5Q01 Eisenhower Avenue 
Alexandria, VA 22333 

1 Br, Robert Sasmor V 

nJ.S . Army Research Institute for 

the Social and Behavioral Sciences 
5001 Eisenhower Avenue . 
Alexandria, VA 22333 



Si '.v ffr* JJoycie : Jihi^d^ ;•':>. :^ •; C-'-^ 
: U . 5 *~Army Research Institute 
5001- Eisenhower Avenue* ^ . ^ 
; Alexandria, VA. 223^3 / 

JL Drv Hilda* Wing / : . / 

fc U,S* Army "Research Institute 
** v - 5001 Eisenhower ;A venues 
■ Alexandria, VA 22335 / 

1 ;l r Dr * iCb bert(: Wi sher ^ ; -> 
: ^U.S . Army Research Institute 
5001 Eisenhower Avenue v 
: - Alexandria, VA 22333 ; 7 



Air Force 



1 Air Force Human Resources Laboratory. 

Brooks Air Force Base, TX ; 78235 - . : - - 

1 .Technical Documents Center " V ^ ' 
Air Force Human Resources Laboratory 
" WPAFB y OH 4 5433 ~ "-' " :; .. "; .4 ' [ 

- u:S ; Air Force Of f ice ot~ X ~ r ^^^r; ~ 

Scientific Research 
Life Sciences Directorate, NL 
Boiling Air Force Base 7 / v 
Washington, DC 20332 , ^ 

"~. 1 Air University Library : " -\ ^ 
AUL/LSE 76/443 . 1 V / 

Maxwell AfB, AL 36112 . ^ 

_ 1 Dr. Earl A* Alluisi T "'""7T^' ~^T^T 

HQ, AFHRL (AFSC) f I 

Brooks Air Force Base, TX 78235 
••*■«'.."" s - 
1 Mr- Raymond Christal 
AFHKj/MOE 

Brodks AFB, TX 78235 

1 Dr. Alfred R. Fregly 

AFOSR/NL ; 1 ; 

Boiling AFB, DC 20332 v . f . : . 



33 



ERIC 



t5- 



Air Force . ^ . . 

1 Dr - Genevi eve Haddad - v " / V i V i 

Program Manager ~ :■;/._/;• 
Life Sci^nc^SMDirictprate : v . v 

. : ." ' ' AFOSR - ; • _ .; -. - 

^BollingA^B, DC _. 20332 , • . _ 

1 Dr . T , M . L ongr idge ■ . ^ 1 ;.- " ' . : 

^ - Williams AFB A£ |5?24. _ , ^ 

vJA T>r - &ogeX-Renne;i^ , ; 

: ; v Air Force Human Resources Laboratory 
• Lowry AFB , CO . 80230 f ; . 

> 1 Dr - Malcolm Ree . - ^ ; f 

AFKBL/MP „ • 

Brooks Air Force Base, TX 78235 

1 LT Tallarigo " : . 
; — — 3 y qq CHTW / TTCiHR j — r - -- -- - v - — - 

Sheppard AFB, TX 76311 

1 ^r * Joseph Yasatuke : 

: . AFHRL/XRT - - - ; , "- 

V Lowry AFB CO 80230 



Department of Defense 



12 Defense Technical Information Center 

•• Attn: TC - - -V "- : " • V 7 -".'" 

4 Cameron Station, Building 5 
Alexandria, VA 22314 



T~ DrV Craig I * Fields — — — — . . v 
.... Advanced Research Projects Agency 
V 1400 Wilson Blvd. 

Arlington, VA 22209 a ■ 

1 Dr. William Graham V • 

Testing Directorate 
iffifCOM/^EPCT-P 

Ft. Sheridan, IL 60037 , ;'--V 



::Depaxtment-?0f Defense - - \ 

• . ■• ;~- .— .-• ■-<■ - : -v" ! > ••-:*•--•- .. .-: -.: _•.•-> • -'N'iTU 

Mrr, Jerry Lehnus , " •". r 

HQMEPCOM 

Attn: MEPCT-P : : : • 

Ft Sheridan, IL 60037 

Military, Assistant ^Ebr Training . 

and Personnel Technology 
Office^ of the Under Secretary of " : 

Defense for Research and Engineering 
Room 3D129,>The Pentagon; , - 

Washington, DC 20301 " ; " ; 

Dr. Wayne Sellmati \:: x ! 
': office of the Assistant ^Secretary - ; , • 
of Defense (MRA&L) ^ J % 

2B269 The Pentagon , . , ; ; ^ 

Washington, DC 20301 • _ J : \ r - r ' : 



Major Jack Thorpe 

DARPA j 2 1 ■ ; L : 

1400 Wilson Blvd. 

Arlington s " VA 22209 



Civilian Agencies 



Dr*. Patricia A. Butler - • 

HIE-B^i • B^id g * , S top // 7 ~-\ v: ' •"; 

1200 19th Street, NW ^, / 

Washington, DC 20208 ; ■• . 

• Dr s Susan Chipman . ^ ;„ 
Learning, and Development ' 
National Institute of Education 
1200 19th Street NW 
Washington, DC 20208 



Dr. Arthur Helmed 7-- —- -~r~—-^ 
724 Brown i ^ •..:>• , .. r 

U.S. Department of Education 
Washington, DC 20208 ; " 

■ '" ■ '.■ " ' "\ . .. .. ■ ' 4 • 

Dr,. Andrew R, Molnar 

Office of' Scientific and Engineering 

Personnel and Education , 
National Science Foundation 

Washington, DC 20550 . ; y-* r-; r"i> 



34 



9 

ERIC 



-6- 



G±vili:an"AgVrtc±es ^ ^ 



Dr . Vera W - Ur £y ; . 

Personnel R & D Center 
Office of Personnel Management 
:190p E Street, IW V / 
Washington, DC 20415 , 

Mr ♦ Thomas A - Warm " 
^U*S » Goas t Guard - Institute ' 
V P*0^ Substation 18 
V OklahOttia Ci ty , OK 73169V^ ^ 

Dr * Frank Withrow V^ ; ^ ; -Zr 
U.&- Of f ice ! of Education '.. 
409 Maryland 'Avenue » SW 
Washington, DC ; 20202 ^ : *; 

Dr . Joseph L. Young Director 
Memory and Cognitive Processes 
Rational Science Foundation 
Washington, DC 20550 



Private Sector 



^ ^DrV^R^'DkrrelX Bock \'y " .---•;- ! =p 

} v . Department of Edueat ion j ; ..' . 

'University of Chicago v'£ : >" 

Chicago, II- .60637 ; _ ; % 

l\x -Dr v Robert B^rennan. : ^ . \€-T^ 

Programs 

P.O. Box 168 / ? 

oiowa^Gity ^ ^A ^52?43 • ^ 

_1 Dr. Glenn 7 Bryan = : = .. - v ; f 

6208 Poe Road - .*.■"■* ' "\ 
■/ ' , Be thesday MD 20817 ^yf'y'r'J: i ; : : : 

1 Dr. Ernes t"R. Cadotte - 
307 St oka ty 

University of Tennessee v 
Knoxyille , TN 37916 "/.- - ; 



Dr. Pat Carpenter 
Department of Psychology 
Carnegie^eXlon^Jniversi^ 
Pittsburgh, PA 15213 



1 Dr. John B. Cajrroll 



1 Dr . . James- Algina 

University of Florida 
Gainesville, FL 32611 

1 *Dr- Patricia Baggett 

Department of Psychology 

A " University of Colorado ; 
Boulder, CO 80309 " 

1 Dr- Isaac Be jar " 

Educational Testing Service) 
Princeton, NJ 08541 

1 Dr . Me nucha Birenbaum 
, School pt Education. 
Tel Aviv University 
/ Tel Aviv, Ramat Aviv 69978 
ISRAEL 



409 Elliott Road • • . . 

Chapel Hill, NC 27514 

Dr* Norman ^Cliff^.; ^.M.^: r 
Department of Psychology 
University of Southern California 
University Park = 
Lpih Angeles, CA 90007 - .. v- ? - -. 

Dr*v Allan M. Collins 

Bolt, Beranek, and Newman, Inc. 

50 Moulton Street 

Cambridge , MA 02138 

Dr* Lynn A# Cooper , 

LRDC ; ■ "> : 

University of Pittsburgh 
3939 0 f Hara Street 
Pittsburgh,, PA 15213 *' 



35 



9 

ERIC 



* P r ivate S ect or 



Dr Hans Crombag • ; ,y 

Education Research Center 
Univers ity of Leydeti 
Boerjhaayelaan 2 : I 
2334 EN Ley den . . _ 
XllE KETHERLAIJDS ; . T r 

; Dr . Da 1 1 pr adad ; Div^i 
Syracu se Uniyers i ty > \ ^ 

' Department of Psychology,--' 
Syracusei NY 33210 : 



X- Dr ; - Susan Ember c son • - --. - - 
; Psychology Department \ 
University of Kansas . ; 
Lawrence, KS 66045 r ^ 

1 ERIC Facility-Acquisitions 
4833 Rugby Avenue 
Bathes da, MP 20014' 



1 Dr. Benjamin A • Fairbank, Jr. 
HcFann-Gray and Associates, Inc. 
5825 Callaghan - - ; -~ 
Suite 225 

San Antonio, TX. 78228 

1 Dr. Leonard Eeldt - . 

Lindquist Center for Measurement 
University of Iowa " 
Iowa City, I A 52242 '. 

1 prof. Donald Fitzgerald 
University of New England 
Armidale, New South Wales 2351 
AUSTRALIA 

4 -/pr «i Dexter .Fletcher — ---- 



Private Sector^" 7 



1 Dr. Janice Gifford/ *_ 
University of Massachusetts 
School of Education 
Amhers t , MA 01002 ~- 

1" Dr .Robert Glaser 
LRDC 

r - University _of: Pittsburgh 
3939 0 f Hara Street 
Pittsburgh > : ?K- 15213 : ; ; 

' 1 Dr . Bert Green [- ' . • • * v,. 
Department of Psychology- i 
.1 ^Johns Hopkins Univeifsity 
Charles and 34th Streets 
fealtimore, MET 21218 V':'^.~' 

1 . Dr . < Ron Hambleton -_" r .:? 

School of Education 
1 Univers Ity of Massachusetts j 
: : Amherst; MA 01002 , •■ 



Dr. Paul Horst ^ 

677 G S treet," #184—" r 

Chula Vis ta , CA 90010 

Dr . Lloyd Humphreys 
LPepar tjaenty ©{. /gf y ^hology 
University of ^Illinois 
Champaign, IL 61820 



WICAT Research Institute 
1875 S- State 1 Street v ; 
Orem, UT 22333 V , 

Dir. John R. Frederiksen 
Bolt, Beranek, and Newman 
50 Mouiton Street- 
Cambridge, MA. 02138 



Dr. Jack Hunter 

2122 Coolidga Street ' 

Lansing, MI 48906 

=. y- t . v = .. 

Dr.^tiuynh Huytih, 
College of Education • ' 
University of South Carolina 
Columbia, SC 29208 

Dr. Douglas H. Jones 
10 Trafalgar Court 
Lawrencevi lie, NJ 08648 



36 

ERIC 



Private S ect or 



F rivate Sect or 



1 Prof -John A. Keats , - ^ 

' Department of Psychology 
*UniverBity of Newcastle ; 
Newcastle, New South Wales :23Q8 
AUSTRALIA ^ fV : V'.-/- //C/^tcV^ 

1 ic, Wiliiam^Koch 4, :^ r 
University of Texas^Austin . 
• He a sur elite n t and ; E va lua t i on Center 
- Austin, XX 78703 ; v 



1 Dr .; Pat, Langley -vv;->_ = : . V. ; 

- The Robotics Institute. 
r v Carnegie^ierion University. 
■;. Pitt sburgh, PA 15213; . 



1 Dr / Alan Xesgolcf • ; " • k _. 

.Learning R & D tenter 
University of Pittsburgh 
3939 ;0*Hara Street 
Pittsburgh, PA 15260 

^Department of Educational Psychology 
~ . 210 : Educat ion- Building -"- - - ■- 

, • University >of Illinois - 

Champaign, IL 6J,801 • • " 

1 Dr. Charles Lewis V . ■ <fS 

" Faculteit Sociale Wetenschappen - - 
Rijksuniversiteit Groningen 

1^ Oude Boteringestraat 2.3 

ti 9712GC Greningen ^ 
• * NETHERLANDS 

I Dr . .Robert Linn 

College of Education \ v 

. University of Illinois 

Urbana , IL 61801 

. I Mr, Phillip Livingston 

'< Systems and Applied Sciences Corporation 
* 68111 Kenilwprth Avenue ; . - : . 

ftiveirdale , MB 20840 * 



I Dr. Robert LocKm^n 

Center for Naval Analysis V 
r 200 North Beauregard Street 
'-. Aiexahdr lav VA V 22311., • ; 

%1 Dr v Frederic H t».6 W t : 

* Educational; Testing ^Brylee "v-V/, £ 
, Princeton, NJ OB^t \ ? . ". ; v. ^ 

. 1 . ; , ' Dr - . J ame s Lums d e A ; - .-.S. : - \"H : 
Departisent: bf .Psycho>gy ... 
-University of Western Australia 
; Nedlands P Wis tern; •Australia, ; 6009 
: "AUSTRALIA -V" : — W~;^tr^ 

1 Dr* Gary Marco " - ; - .....^ 
' . Stop 31-E . ■ ''r ' \; ; :' >.•'; •• 

Educational Testing Service 
Princeton, NJ 085A1 / ! 



pr. Scott' Maxwell v , 
Department of Psychology 
University of —Notre Dane 
Ndtre Dame , : IK A 6 556 .:;.;:, 



1 . Dr . Samue 1-T Mayo- ^ ^ ; 

Loyola University of Chicago 
820 North Michigan Avenue ; _ 
Chicago, It 60611 

1 Mr. Robert McKinley : 
American College fasting Programs 
P.O. Box 168 -. ...... 

Iowa City, IA 52243 - 

1 Dr. Robert Mislevy 
; 711 Illinois Street ' , ; * \ 
Geneva, IL 60134 : \ . 

1 Dr . Allen Munro -_ — ^ ^ ~- — — - 
Behavioral TechnolciiyLaboratorie 

1845 Elena Avenue, fourth Floor 
Redondb Beach , CA 90277 



-9- 



Private I, S ecicsMi 



- 1 Dr. Alan Nicewande r : - , ^ - ^ - - - 
University of Oklahoma : v ; 

Department of Psychology > y --y/.-y 
Oklahoma City, OK 73069. ; _ . 

J 1 Dr • Donald A. Gorman ■'; 

Cognitive Science , C«r015 . 
University of Calif ora^ 

La -Jo 11a , ^CA ; 92093 ; ; y ; ; y V : 

1 p r * " Me 1 vi n R 5 N o vi ck< y 7 ; * 

- 35 6 L i nd„q ui s t Cent e r f o r Measur emen t 
y University of Iowa - , 

:^ \;. Iowa -City, lA 52242 v - y ; - y 

1 Br* James Olson y; : • y 

WICAT, Inc. "";'v'"-;'.-'« ; 
, * 1875 SY State Street / ' • ' 
Orem; UT 84057 



Dr . ~ THpmas R£3^nold$ ~ y- , 
University of Texas, Dallas 
Marke t ing. D ej^rtment ; 
P.O, Box 688 - . ' 

fSSt chards da,^ri§3 - 75080 y y: 



Dr ^ yAndr ew^R? 
American Inst 
1055 Tbptnas J 
Washington^ • D' 



Dr. Ernst Z . SRothkopf ••■ 
Bell l* abbra fceSfeies • 
Murray ri^l; »M • 07974 

Dr * Lawrence ^Sudner . 

403 Elm'-AvettU"e;;-^y^:::?;";;"'.y; 
Takoma: Park, 20012 



itu-r.es /for Research 
ef ferson S t v i NU _ 
20007 , • y. 



Wayne M - Patience ^ 
hfae'r i c an jC bun c j. 1 ; ori ca t ioa 
GED Testing Service, Suite 20 
-One Dupont Circle , NW -v \^ 
Washington-, DC 20036 

Dr. James A. Paulson 
Portland State University 
P.O- Box 751 . ... .., . . ^ 

Portland, OR 97 207 r * - 

Dr. James W. Pellegrino 
Univeristy of California, 

Santa Barbara -« 
Department of Psychology 
Santa Barbara, CA 93106 ; 

= '. ■ ■ ■•' / . • 

Dr.. Mark D. Reckase 
ACT ' : ._ v ; y,,y ; -y , 

P.O* Box 168 
Iowa City, IA 52243 

Dr* ; Lauren ;Reinick ■ 

LRDC - ■ 
University of; Pittsburgh : V ; 
3939 0 ^Hara S treat y ■ V' 
Pittsburgh, PA 45261 / / 



Dr. J - Ryan/ 
Department *c?f" 
Univftrsity 



^ Education 
Slou th Carolina 



Prof Fumika Same jima . 
Department <pP Psychology 
University Tennessee 
Knoxville, Tt^*; 37916 



pr. Walter- $ 
Psychology P 
603 E» Danie 
Champaign, J 



hneider. 
artmerit 



61820 



Dr . Lowell £c==rhOer ' 
Psychologies^- and Quantitative 

> Founda t i £tr=is - -i V. "y r . 

College of B^3ucat ion ' 

University Iowa . . . 

Iowa City, 52242 > . 

CrV Robert J— Seidel V ; ' 
Instructiona^l Technology Group 
HUMRRO . 
300 N^ WashiBmgt on Street 
Alexandria:, 22314 ; y \ 



ERIC 



Priva te Sect or 



/¥riya te S ect or- 



1 . Dr : . Kazuo Shigema su - * — - _ . ; : ; 

University of Tohoku A 
S Department of Educational Psycliology 
Kawauchi 9 Sendai 980 v ; . ; " 

JAPAN ,„-.-;_ 

1 Dry Eowin Shirkey ^ 
> Department of Psychology ; " A ; " 
University of Central Florida 
V- - • Orlando, PL "3281 <6 ; . - ; " . \ . ". 

1 ^rpmias SlE ; . /' vv; •:/ \;V;< :0 
: : Center for Naval -Analysis .--."\ ■ 
j ; 200 ^or tlx 6eaureg^rd% v S tree t - ?r":; 
Alexandria, VA> 22311 ' • V ; ^ 



r lJr^ H,, Wallace Sinaiko ^ 
; "Pxogram director m 

Manpower Research arid Advisory Se 
' Smithsonian institution^ \ 

801 North Pitt Street 

Alexandria, VA 22314 

- , ? - School of E ducat Ibn- - -7- ™ 
- Stanford University " V 
-Stanford, CA 94305 

1 Dir. Kathryn T, Spoehr . : 
Psychology Department ; 

r^/::B*owj»^ Un ive rs i t y ^3 -rr r r-":r * 

Providence, RI 02912 - ; 

#1 Dr. Robert Sternberg 
■ i Department of Psychology 
, • Yale University s . ; ■ 

\ Box 1 1A, Yale Station 

^ ' - New Hayeri> CT 06520 - 

1 ? Dr, Peter/ Stblof f , - 

Center- f drNava 1 Analysis 
• 20a North *Beauregard Street 
Alexandri&PVA 22311 



ri[ices- 



iS^tir'. William \Steu;t ^: ; _ 

University pi Illinois v l . 

Ifepar tme rit ; oi Ma thema t ic s r 
- Urbana , IL 61801 : ; 

l ;Dr, Patrick -Suppes . j 
v Institute for Matheinaticai Studies ' 
■ in , th e. $dcial S ciences^: : . 
- Stanford^ University 
V Stanford; CA 94305 _ 

1 ) Dr . Hariharan SwaminathanV - _•; '. 
< ; • La bora tory of Psychbme trie: and : v .;' • 
t ^ -Evaluation; Regeareh • : > y - . ; ;J • 
School of Education 
University of Massacuusetts 
Amherst, MA 01003 ; ; \ v' : > ' 

1 Dr w Kikumi Tatsuoka " : • '• 

, Computer Based Education Research; 
- • . ' • Laboratory _ ■= , . / 

25 2 Engineering Resear cfe ; l,abora tpry 
.:' " Unive rs ity of Illinois >v ~ . - 
; ^Urbana:, . IL.^ 618.0 k 

1 Dr. Haur ice Tatsuoka 
220 .Education Building 
1310 S* Sixth Street 
.Champaign, II 61820 

1 . Dr .. . David Thissen 



Department 61 Psycho logyp'^t •i"': W: ".''""'-- = 
University of Kansas V = 

Lawrence, KS 6604-4 ; : : } 1 / 

Dr* Douglas Towne 

Un ive rs ity of Sou them Calif o mia 
Behavioral Technology Labs v 
1845 S* Elena Avenue . 
Redondo Beach, CA 90277 " 



Dr. Robert Tsutakawa 
Department of Statistics 
University of Missouri 
-Columbia, M0 65201 



39 



ERIC 



1 Dr..rV . "R. R , Uppulur i - \ 

Union Car ta^ide Corporation ; - 
Nuclear D ^.vision r - 

•P*0 . Bo^."^5f , ^ v - - 

Oak Riiga X, TN : 37830 : - ■'■ 

1 Dr- DaVidl Vale ".; •• • 
;As ses smet»^Jt Systems Corporation 
2233 ^iv^irsity Avanu^ 4 : / : 
i Suite 3 IC*^ , : . 

St, Paul ^ MT 55114 / , 

1 \ Dr.. /Kurt: ""Van Lehn 

^CeroK PAft^C, : - -v : ." '". ■•" 

^333 3 C oy te Hill Road . 

/Mo .Mt£i 5 : CA 94304 



T' Dr- How#i 
Edudatiot 
Princeton 



-cl *Wainer 

Testing Service 
KJ 08541 r 



-1" 



Dr * Rand R- Wilcox - 
University of Southern California 
department" of P sy etiology V 
Xos .Angeles ,; €A ;,90pp7 ^ ; . v ; 

Br v Wolfgang WildgruDe - : :-'• 

Streitkraef teamt : : 

Bo ? 20 50 03 - ■;; 

^5300 Bonn 2 • ; : . : : \ "? . \ - \ , 

we si gebmAny. / ; J .. ' / 

• V." \- V " X:./.. > . Y X- ./:: 'Y : f 

^Dr . Bruce' Williams \-y. s - • : V > v„ 
Department of Educational Psychology 
University of Illinois y.; : YY; 
Urbanay II* -61801 • : " ; 

Dr. Wendy Yen ,;-^\ v.a 

CTB/Mcttraw^Hill ' / ; Yw; ; ; V^vY* 

Del Monte Research Park ' \ '' ■-' , " ' - . 
Monterey, CA 93940 ." . C T : 



a:lDr^i;Mic*i^^ 

Departmaizrit of Educational Psychology 
Univers it^iy of Wisconsin 
'Milwaukee, WI 53201 ; ■ '- 



1 Dr.v Eria 
~ HUMKRO. 

Alexander 



Waters* 

Washingt on , . 
a, VA 2231$ 



.lvDr* . FhylIB_is -Weaver 
2979 Alexis Drive v,/*' 
Palo Alto, CA 94304 



1 



DrV Davie 
M660 Ell: 
UniyersH 

45-Easfa^I 



L JY Weiss ; ;.. - 
Lott Hall y- ; f , 
ly of Minnes.ota 1 
Live r Road- Y!1j^ 
Lis, M 55455 i 



'fDr:^ : Keitfcri T . Wescour t 
Perdep tt^jnics fj! Inc • ; 
545 ^Widd^iWf ieid Road 

Suite 140 f 

Menld Paack, CA 9<025 



